Enhancing Protocol Evaluation Through Semantic Modification of Local Benchmarks

A computer based evaluation is essential to allow us to assess students’ protocols without human intervention in R SAT (Reading Strategy Assessment Tool). Word matching and Latent Semantic Analysis (LSA) approaches were explored for use in the evaluation of protocols and. experimentally it was found that word matching alone outperformed LSA or LSA in combination with word matching. Why does LSA not work well with the R SAT protocol evaluation? This paper demonstrates that modifying the benchmarks on which it is based can indeed benefit its performance. A variety of local benchmark modifications are investigated and results are compared against human expert. The preliminary results show that modified local benchmarks improve the R SAT protocol evaluation on local bridging strategy: 0.1 increase in correlation and 8% increase in percent agreement between original benchmarks and modified benchmarks.